{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Automatic  parameter tuning.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xRKJmLIwkoT7"
      },
      "source": [
        "[参考链接 - datawhale - 机器学习调参自动优化方法 ](https://mp.weixin.qq.com/s/qFhRjBBXXwCKnyPIh5GSiQ) <br>\n",
        "[参考链接 - sklearn - Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html#grid-search)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kbiWBR-KlF_S"
      },
      "source": [
        "### 一、网格搜索(Grid Search)\n",
        "\n",
        "网格搜索是暴力搜索，在给定超参搜索空间内，尝试所有超参组合，最后搜索出最优的超参组合。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "06GWgg8oj-V8"
      },
      "source": [
        "from sklearn import svm, datasets\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "import pandas as pd\n",
        "\n",
        "iris = datasets.load_iris()\n",
        "\n",
        "parameters = {'kernel': ('linear', 'rbf'), 'C':[1, 10]}  # 定义超参搜索空间\n",
        "\n",
        "svc = svm.SVC()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cD_GHeoImDho",
        "outputId": "eb742461-f3a0-472b-d674-3df11ed94cbf"
      },
      "source": [
        "# 网格搜索\n",
        "clf = GridSearchCV(estimator = svc,\n",
        "          param_grid=parameters,  # 超参搜索空间，即超参数字典\n",
        "          scoring='accuracy',   # 在交叉验证中使用的评估策略。\n",
        "          n_jobs=-1,  # 并行任务数，-1为使用所有CPU\n",
        "          cv=5)   # 决定采用几折交叉验证\n",
        " \n",
        "clf.fit(iris.data, iris.target)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=5, error_score=nan,\n",
              "             estimator=SVC(C=1.0, break_ties=False, cache_size=200,\n",
              "                           class_weight=None, coef0=0.0,\n",
              "                           decision_function_shape='ovr', degree=3,\n",
              "                           gamma='scale', kernel='rbf', max_iter=-1,\n",
              "                           probability=False, random_state=None, shrinking=True,\n",
              "                           tol=0.001, verbose=False),\n",
              "             iid='deprecated', n_jobs=-1,\n",
              "             param_grid={'C': [1, 10], 'kernel': ('linear', 'rbf')},\n",
              "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
              "             scoring='accuracy', verbose=0)"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ubL2OsP0mem0",
        "outputId": "354fb0cf-d598-41b0-d42a-dc2b34bebf1c"
      },
      "source": [
        "print('详细结果:\\n', pd.DataFrame.from_dict(clf.cv_results_))\n",
        "print('最佳分类器:\\n', clf.best_estimator_)\n",
        "print('最佳分数:\\n', clf.best_score_)\n",
        "print('最佳参数:\\n', clf.best_params_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "详细结果:\n",
            "    mean_fit_time  std_fit_time  ...  std_test_score  rank_test_score\n",
            "0       0.001554      0.000904  ...        0.016330                1\n",
            "1       0.001253      0.000097  ...        0.021082                4\n",
            "2       0.000947      0.000097  ...        0.038873                3\n",
            "3       0.000866      0.000062  ...        0.016330                1\n",
            "\n",
            "[4 rows x 15 columns]\n",
            "最佳分类器:\n",
            " SVC(C=1, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,\n",
            "    decision_function_shape='ovr', degree=3, gamma='scale', kernel='linear',\n",
            "    max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
            "    tol=0.001, verbose=False)\n",
            "最佳分数:\n",
            " 0.9800000000000001\n",
            "最佳参数:\n",
            " {'C': 1, 'kernel': 'linear'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FH4gFiyMnjp1"
      },
      "source": [
        "### 二、随机搜索(Randomized Search)\n",
        "&emsp; 随机搜索是在搜索空间中采样出超参组合，然后选出采样组合中最优的超参组合。\n",
        "\n",
        "\n",
        "&emsp; 如果目前我们要搜索两个参数，但参数A重要而另一个参数B并没有想象中重要，网格搜索9个参数组合(A, B)，而由于模型更依赖于重要参数A，所以只有3个参数值是真正参与到最优参数的搜索工作中。反观随机搜索，随机采样9种超参组合，在重要参数A上会有9个参数值参与到搜索工作中，所以，**在某些参数对模型影响较小时，使用随机搜索能让我们有更多的探索空间**。\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wNQQcjqHnLv0"
      },
      "source": [
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "from scipy.stats import uniform\n",
        "\n",
        "distributions = {'kernel':['linear', 'rbf'], 'C':uniform(loc=1, scale=9)}  # 定义超参搜索空间\n",
        "   # uniform 均匀连续分布"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OMa3LPazrMBR",
        "outputId": "56e7c065-7d4f-4a03-fe56-fe25d8303069"
      },
      "source": [
        "# 随机搜索\n",
        "clf = RandomizedSearchCV(estimator= svc, \n",
        "              param_distributions=distributions,\n",
        "              n_iter = 4,\n",
        "              scoring='accuracy',\n",
        "              cv=5,\n",
        "              n_jobs=-1,\n",
        "              random_state = 2021\n",
        "              )\n",
        "clf.fit(iris.data, iris.target)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomizedSearchCV(cv=5, error_score=nan,\n",
              "                   estimator=SVC(C=1.0, break_ties=False, cache_size=200,\n",
              "                                 class_weight=None, coef0=0.0,\n",
              "                                 decision_function_shape='ovr', degree=3,\n",
              "                                 gamma='scale', kernel='rbf', max_iter=-1,\n",
              "                                 probability=False, random_state=None,\n",
              "                                 shrinking=True, tol=0.001, verbose=False),\n",
              "                   iid='deprecated', n_iter=4, n_jobs=-1,\n",
              "                   param_distributions={'C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f930480e590>,\n",
              "                                        'kernel': ['linear', 'rbf']},\n",
              "                   pre_dispatch='2*n_jobs', random_state=2021, refit=True,\n",
              "                   return_train_score=False, scoring='accuracy', verbose=0)"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h9IC-i5fr5hh",
        "outputId": "2741883d-7471-4581-cd73-eb27ddef0201"
      },
      "source": [
        "print('详细结果:\\n', pd.DataFrame.from_dict(clf.cv_results_))\n",
        "print('最佳分类器:\\n', clf.best_estimator_)\n",
        "print('最佳分数:\\n', clf.best_score_)\n",
        "print('最佳参数:\\n', clf.best_params_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "详细结果:\n",
            "    mean_fit_time  std_fit_time  ...  std_test_score  rank_test_score\n",
            "0       0.001050      0.000118  ...        0.016330                1\n",
            "1       0.001490      0.001139  ...        0.026667                3\n",
            "2       0.001409      0.000321  ...        0.016330                3\n",
            "3       0.001649      0.000689  ...        0.016330                1\n",
            "\n",
            "[4 rows x 15 columns]\n",
            "最佳分类器:\n",
            " SVC(C=6.453804509266643, break_ties=False, cache_size=200, class_weight=None,\n",
            "    coef0=0.0, decision_function_shape='ovr', degree=3, gamma='scale',\n",
            "    kernel='rbf', max_iter=-1, probability=False, random_state=None,\n",
            "    shrinking=True, tol=0.001, verbose=False)\n",
            "最佳分数:\n",
            " 0.9866666666666667\n",
            "最佳参数:\n",
            " {'C': 6.453804509266643, 'kernel': 'rbf'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUnAYfoktB07"
      },
      "source": [
        "### 三、贝叶斯优化(Bayesian Optimization)\n",
        "[看原推文解释](https://mp.weixin.qq.com/s/qFhRjBBXXwCKnyPIh5GSiQ)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oVKwk5udsFF5"
      },
      "source": [
        "from sklearn.model_selection import cross_val_score\n",
        "from hyperopt import hp, fmin, tpe, space_eval"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I9Iji0vUv9GO"
      },
      "source": [
        "# step1: 定义目标函数\n",
        "\n",
        "def objective(params):\n",
        "  # 初始化模型并交叉验证\n",
        "  svc = svm.SVC()\n",
        "  cv_scores = cross_val_score(svc, iris.data, iris.target, cv=5)\n",
        "  \n",
        "  # 返回loss = 1 - accuracy (loss必须被最小化)\n",
        "  loss = 1 - cv_scores.mean()\n",
        "  return loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4LekvVRvxAcB"
      },
      "source": [
        "# step2: 定义参数搜索空间\n",
        "\n",
        "space = {'kernel': hp.choice('kernel', ['linear','rbf']),\n",
        "         'C': hp.uniform('C', 1, 10)}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SYklE1f8xbti",
        "outputId": "d552e73a-0227-4019-b6f2-50f736ac5b68"
      },
      "source": [
        "# step3: 在给定超参搜索空间下，最小化目标函数\n",
        "\n",
        "best = fmin(objective, space, algo=tpe.suggest, max_evals=100)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100%|██████████| 100/100 [00:01<00:00, 79.13it/s, best loss: 0.03333333333333344]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bk4QvyFsxr98",
        "outputId": "155ac4e7-3dfd-4e8a-90e4-5c6b0f3fc504"
      },
      "source": [
        "print(best)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'C': 5.093638570678445, 'kernel': 1}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ticgKSkdyLxL"
      },
      "source": [
        "### 四、Hyperband\n",
        "\n",
        "Hyperband本质上是随机搜索的一种变种，它使用早停策略和Sccessive Halving算法去分配资源，结果是Hyperband能评估更多的超参组合，同时在给定的资源预算下，比贝叶斯方法收敛更快。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-gvaWbV7xzvm"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}